Tree-Structured Trajectory Encoding for Vision-and-Language Navigation

Over the past few years, research on vision-and-language navigation (VLN) has made tremendous progress. Many previous works attempted to improve performance from different aspects like training strategy, data augmentation, pre-training, etc. This work focuses a rarely-explored aspect in VLN, namely trajectory organization and encoding during navigation. Most of existing state-of-the-art VLN models adopt vanilla sequential strategy for trajectories. Such takes whole as single sequence estimate current state, no matter whether agent moved smoothly or perhaps mistakes backtracked past. We show that may largely lose this kind fine-grained structure trajectory, which could hamper later state estimation decision making. In order solve problem, proposes novel tree-structured strategy. The is organized tree rooted starting position, encoded using our Tree-Transformer module fully extract historical information. Besides, spatial topology be easily embedded tree, we further design tree-based action space allow making long-range error-correction one decision. implement holistic based cross-modal transformer train it with newly-proposed Tree-nDTW reward. On benchmark dataset R2R, model achieves surpassing success rate (SR) 68% val-unseen 66% test. conduct extensive ablation studies analyses provide more insights effectiveness designs.